Kid metric added #435

minettekaum · 2025-11-11T16:17:17Z

Description

Added KID (Kernel Inception Distance) metric

Related Issue

Fixes #(issue number)

Type of Change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How Has This Been Tested?

The KID metric implementation has been tested with the following test suite:

Main KID Test (test_kid):
- Tests KID metric functionality on both CPU and CUDA devices
- Uses LAION256 dataset with 4 batches of images
- Validates that KID returns ~0 when comparing identical images (abs < 0.25)
- Ensures proper handling of subset_size parameter (must be smaller than number of samples)
- Checks for NaN values to ensure numerical stability
- Test cases: test_kid[LAION256-cpu] and test_kid[LAION256-cuda] - both PASSED
Call Type Tests (test_check_call_type):
- Validates KID works correctly in both "single" and "pairwise" evaluation modes
- Ensures proper call type conversion (gt_y for single, pairwise_gt_y for pairwise)
- Test cases: test_check_call_type[single-kid] and test_check_call_type[pairwise-kid] - both PASSED
All Torch Metrics Tests:
- Ran full test suite: uv run pytest tests/evaluation/test_torch_metrics.py -v
- Result: 41 passed, 7 skipped (CUDA tests on macOS), 0 failed
- Confirmed no regressions in existing metrics

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

Additional Notes

begumcig

Thank you so much for implementing KID, Minette, it already looks amazing! 💜 I just left one small comment: I think we could handle the special case in compute() a bit more cleanly (maybe by adding a kid_compute to the TorchMetrics enum?), so that the compute() function stays as generic as possible. Other than that, it’s already looking great and pretty much ready to go! 💜💜

begumcig · 2025-11-11T16:54:43Z

src/pruna/evaluation/metrics/metric_torch.py

        result = self.metric.compute()

+        # Handle KID which returns a tuple (mean, std)
+        if self.metric_name == "kid" and isinstance(result, tuple) and len(result) == 2:


I don't really have a suggestion for this yet but I think we should not do conditional statements in the compute function based on the metric name, as this can get messy quite easily, how do you feel?

Very good point, and I agree. Having conditionals in compute() based on the metric name can get messy fast. So I cleaned it up a bit:

Took out the if self.metric_name == "kid" check from compute().

Added a kid_compute function to handle KID’s tuple return.

Updated the TorchMetrics enum so KID has a 4th element for kid_compute (others stay as 3).

Now compute() just uses self.compute_fn from the enum if it’s there, otherwise falls back to the default self.metric.compute().

So now the compute() method is generic and clean, no metric-specific conditionals. Each metric can have its own compute logic in the enum, just like how we handle the update functions.

What do you think, does this look like a good direction?

sharpenb

This looks good! Left a single comment

src/pruna/evaluation/metrics/metric_torch.py

github-actions · 2025-11-25T00:07:04Z

This PR has been inactive for 10 days and is now marked as stale.

begumcig

Looks super good Minette! I made a small comment about the blank line but feel free to merge! Amazing job you are a queen 👸

tests/evaluation/test_torch_metrics.py

github-actions · 2025-12-08T00:08:40Z

This PR has been inactive for 10 days and is now marked as stale.

…oc>=1.6.0` and `ty==0.0.1a21` for compatibility. (#417) * pin torchao version. * regenerate uv lock file as well. * fix ty dep indent * update dep for sphinx as well. * fix syntax error. * update the lock file as well. * rm linting rule. * revert. * update transformer version for janus, try not to break ci. * update the lock file as well. * pin ty version for linting. * break deps for logs. * add comment for torchao dep. * add new lock file. * pin ty version as in lock file. * pin numpydoc for jinja version conflict. * unpin transformers version. * update lockfile as well. * Kid metric added (#435) * feat/kid-metric and updated docs * fixed typo * fixed linting error * moving KID logic to enum * fixing linting error * fixed another linting error * fixing linting error * delete uv lock as per request. * add back the uv lock file. * add back the uv lock file. * regenerate uv lock file as well. * update the lock file as well. * update the lock file as well. * pin ty version for linting. * add new lock file. * pin numpydoc for jinja version conflict. * update lockfile as well. * update lock file for last time. --------- Co-authored-by: Minette Kaunismäki <[email protected]>

* feat/kid-metric and updated docs * fixed typo * fixed linting error * moving KID logic to enum * fixing linting error * fixed another linting error * fixing linting error

…oc>=1.6.0` and `ty==0.0.1a21` for compatibility. (#417) * pin torchao version. * regenerate uv lock file as well. * fix ty dep indent * update dep for sphinx as well. * fix syntax error. * update the lock file as well. * rm linting rule. * revert. * update transformer version for janus, try not to break ci. * update the lock file as well. * pin ty version for linting. * break deps for logs. * add comment for torchao dep. * add new lock file. * pin ty version as in lock file. * pin numpydoc for jinja version conflict. * unpin transformers version. * update lockfile as well. * Kid metric added (#435) * feat/kid-metric and updated docs * fixed typo * fixed linting error * moving KID logic to enum * fixing linting error * fixed another linting error * fixing linting error * delete uv lock as per request. * add back the uv lock file. * add back the uv lock file. * regenerate uv lock file as well. * update the lock file as well. * update the lock file as well. * pin ty version for linting. * add new lock file. * pin numpydoc for jinja version conflict. * update lockfile as well. * update lock file for last time. --------- Co-authored-by: Minette Kaunismäki <[email protected]>

* feat/kid-metric and updated docs * fixed typo * fixed linting error * moving KID logic to enum * fixing linting error * fixed another linting error * fixing linting error

…oc>=1.6.0` and `ty==0.0.1a21` for compatibility. (#417) * pin torchao version. * regenerate uv lock file as well. * fix ty dep indent * update dep for sphinx as well. * fix syntax error. * update the lock file as well. * rm linting rule. * revert. * update transformer version for janus, try not to break ci. * update the lock file as well. * pin ty version for linting. * break deps for logs. * add comment for torchao dep. * add new lock file. * pin ty version as in lock file. * pin numpydoc for jinja version conflict. * unpin transformers version. * update lockfile as well. * Kid metric added (#435) * feat/kid-metric and updated docs * fixed typo * fixed linting error * moving KID logic to enum * fixing linting error * fixed another linting error * fixing linting error * delete uv lock as per request. * add back the uv lock file. * add back the uv lock file. * regenerate uv lock file as well. * update the lock file as well. * update the lock file as well. * pin ty version for linting. * add new lock file. * pin numpydoc for jinja version conflict. * update lockfile as well. * update lock file for last time. --------- Co-authored-by: Minette Kaunismäki <[email protected]>

* feat/kid-metric and updated docs * fixed typo * fixed linting error * moving KID logic to enum * fixing linting error * fixed another linting error * fixing linting error

…oc>=1.6.0` and `ty==0.0.1a21` for compatibility. (#417) * pin torchao version. * regenerate uv lock file as well. * fix ty dep indent * update dep for sphinx as well. * fix syntax error. * update the lock file as well. * rm linting rule. * revert. * update transformer version for janus, try not to break ci. * update the lock file as well. * pin ty version for linting. * break deps for logs. * add comment for torchao dep. * add new lock file. * pin ty version as in lock file. * pin numpydoc for jinja version conflict. * unpin transformers version. * update lockfile as well. * Kid metric added (#435) * feat/kid-metric and updated docs * fixed typo * fixed linting error * moving KID logic to enum * fixing linting error * fixed another linting error * fixing linting error * delete uv lock as per request. * add back the uv lock file. * add back the uv lock file. * regenerate uv lock file as well. * update the lock file as well. * update the lock file as well. * pin ty version for linting. * add new lock file. * pin numpydoc for jinja version conflict. * update lockfile as well. * update lock file for last time. --------- Co-authored-by: Minette Kaunismäki <[email protected]>

minettekaum requested review from begumcig, sdiazlor and sharpenb November 11, 2025 16:17

minettekaum changed the title ~~Feat/kid metric~~ Kid metric added Nov 11, 2025

begumcig requested changes Nov 11, 2025

View reviewed changes

sharpenb approved these changes Nov 14, 2025

View reviewed changes

src/pruna/evaluation/metrics/metric_torch.py Show resolved Hide resolved

github-actions bot added the stale label Nov 25, 2025

begumcig approved these changes Nov 27, 2025

View reviewed changes

tests/evaluation/test_torch_metrics.py Show resolved Hide resolved

github-actions bot removed the stale label Nov 28, 2025

github-actions bot added the stale label Dec 8, 2025

minettekaum added 7 commits December 10, 2025 16:11

feat/kid-metric and updated docs

63d0926

fixed typo

e0d1707

fixed linting error

ca96cae

moving KID logic to enum

beb9a59

fixing linting error

496fa93

fixed another linting error

84a193f

fixing linting error

2acf02e

minettekaum force-pushed the feat/kid-metric branch from 2620cea to 2acf02e Compare December 10, 2025 15:13

minettekaum merged commit 7d11666 into main Dec 10, 2025
6 checks passed

Marius-Graml pushed a commit that referenced this pull request Jan 19, 2026

Kid metric added (#435)

a20e323

* feat/kid-metric and updated docs * fixed typo * fixed linting error * moving KID logic to enum * fixing linting error * fixed another linting error * fixing linting error

Marius-Graml pushed a commit that referenced this pull request Jan 19, 2026

Kid metric added (#435)

7d9bdaf

* feat/kid-metric and updated docs * fixed typo * fixed linting error * moving KID logic to enum * fixing linting error * fixed another linting error * fixing linting error

Marius-Graml pushed a commit that referenced this pull request Jan 19, 2026

Kid metric added (#435)

07173d8

* feat/kid-metric and updated docs * fixed typo * fixed linting error * moving KID logic to enum * fixing linting error * fixed another linting error * fixing linting error

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kid metric added #435

Kid metric added #435

Uh oh!

minettekaum commented Nov 11, 2025

Uh oh!

begumcig left a comment

Uh oh!

begumcig Nov 11, 2025

Uh oh!

minettekaum Nov 12, 2025

Uh oh!

sharpenb left a comment

Uh oh!

Uh oh!

github-actions bot commented Nov 25, 2025

Uh oh!

begumcig left a comment

Uh oh!

Uh oh!

github-actions bot commented Dec 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Kid metric added #435

Kid metric added #435

Uh oh!

Conversation

minettekaum commented Nov 11, 2025

Description

Related Issue

Type of Change

How Has This Been Tested?

Checklist

Additional Notes

Uh oh!

begumcig left a comment

Choose a reason for hiding this comment

Uh oh!

begumcig Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

minettekaum Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

sharpenb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Nov 25, 2025

Uh oh!

begumcig left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Dec 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants